Implement minimal built-in checks for Ibis backend #1885

deepyaman · 2024-12-25T18:40:01Z

This implements the built-in check machinery and a single validation (equal_to). In future PRs, I (or somebody else) can:

Add more built-in checks
Add more datatypes and make sure they work with the built-in checks

codecov · 2024-12-25T18:41:28Z

Codecov Report

Attention: Patch coverage is 94.73684% with 1 line in your changes missing coverage. Please review.

Project coverage is 92.91%. Comparing base (ffe6c55) to head (34daca5).
Report is 56 commits behind head on ibis-dev.

Files with missing lines	Patch %	Lines
pandera/api/ibis/components.py	83.33%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           ibis-dev    #1885      +/-   ##
============================================
+ Coverage     92.76%   92.91%   +0.14%     
============================================
  Files           133      134       +1     
  Lines          9705     9722      +17     
============================================
+ Hits           9003     9033      +30     
+ Misses          702      689      -13

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: Deepyaman Datta <[email protected]>

cosmicBboy

thanks @deepyaman, this looks great!

Do you plan on implementing the other built-in checks in a separate PR? If so we can go ahead and merge this

deepyaman · 2025-01-13T20:45:55Z

thanks @deepyaman, this looks great!

Do you plan on implementing the other built-in checks in a separate PR? If so we can go ahead and merge this

Yep, will do in a separate PR!

* Implement minimal built-in checks for Ibis backend Signed-off-by: Deepyaman Datta <[email protected]> * Implement `Column` validation for the Ibis backend Signed-off-by: Deepyaman Datta <[email protected]> * Promote check object to table during preprocessing Signed-off-by: Deepyaman Datta <[email protected]> * Remove extraneous fixture for backend registration Signed-off-by: Deepyaman Datta <[email protected]> * Resolve lint (unused imports, undefined variables) Signed-off-by: Deepyaman Datta <[email protected]> * Partially standardize docstrings of builtin checks Signed-off-by: Deepyaman Datta <[email protected]> * Fix the `preprocess` docstrings copied from pandas Signed-off-by: Deepyaman Datta <[email protected]> * Format pandera/backends/ibis/checks.py using Black Signed-off-by: Deepyaman Datta <[email protected]> --------- Signed-off-by: Deepyaman Datta <[email protected]>

* Implement minimal built-in checks for Ibis backend Signed-off-by: Deepyaman Datta <[email protected]> * Implement `Column` validation for the Ibis backend Signed-off-by: Deepyaman Datta <[email protected]> * Promote check object to table during preprocessing Signed-off-by: Deepyaman Datta <[email protected]> * Remove extraneous fixture for backend registration Signed-off-by: Deepyaman Datta <[email protected]> * Resolve lint (unused imports, undefined variables) Signed-off-by: Deepyaman Datta <[email protected]> * Partially standardize docstrings of builtin checks Signed-off-by: Deepyaman Datta <[email protected]> * Fix the `preprocess` docstrings copied from pandas Signed-off-by: Deepyaman Datta <[email protected]> * Format pandera/backends/ibis/checks.py using Black Signed-off-by: Deepyaman Datta <[email protected]> --------- Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]>

…ed pyarrow failed cases (#2036) * resolve 2035 Signed-off-by: Andrej Iring <[email protected]> * change astype to apply Signed-off-by: Andrej Iring <[email protected]> * bugfix: PANDERA_VALIDATION_ENABLED=False should disable validation (#2028) * bugfix: PANDERA_VALIDATION_ENABLED=False should disable validation Signed-off-by: Niels Bantilan <[email protected]> * fix tests Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Expect Python slice index errors after Python 3.10 (#2033) * Expect Python slice index errors with pandas again Seen in https://github.com/unionai-oss/pandera/actions/runs/15526426636/job/43706977542?pr=2030 Signed-off-by: Deepyaman Datta <[email protected]> * Expect Python slice index errors after Python 3.10 Signed-off-by: Deepyaman Datta <[email protected]> * Fix the version check to not include Python 3.10.x Signed-off-by: Deepyaman Datta <[email protected]> --------- Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * resolve failing pyspark tests Signed-off-by: Andrej Iring <[email protected]> * add new test Signed-off-by: Andrej Iring <[email protected]> * format code using black Signed-off-by: Andrej Iring <[email protected]> * Add `DataFrameModel`, `DataFrameSchema` for `ibis` Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Refactor Ibis `DataFrameSchema` to extend pandas's Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Add code for basic `Column`, and stub more modules Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Add Ibis's parsing, validation, and error backends Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Implement stub to validate schema component checks Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Implement validation for floating types Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Inherit getter, only override setter, for `.dtype` Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * implement column, add basic unit tests for Ibis data types Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Add `ibis` extra and regenerate requirements files Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Re-enable Python equivalents for `int` and `float` Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Do not test Ibis below py3.9, add DuckDB for tests (#1773) * Restore accidentally-deleted use of "breakpoint()" Signed-off-by: Deepyaman Datta <[email protected]> * Swap `types-pkg_resources` with `types-setuptools` (#1779) * Swap `types-pkg_resources` with `types-setuptools` Signed-off-by: Deepyaman Datta <[email protected]> * Update the expected error message to name variable Signed-off-by: Deepyaman Datta <[email protected]> * Fix expected outputs for pandas-stubs 2.2.2.240807 Signed-off-by: Deepyaman Datta <[email protected]> * Update CI configuration to filter right pandas ver Signed-off-by: Deepyaman Datta <[email protected]> --------- Signed-off-by: Deepyaman Datta <[email protected]> * Do not test Ibis below py3.9, add DuckDB for tests Signed-off-by: Deepyaman Datta <[email protected]> --------- Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Support specifying and validating optional columns (#1762) * Test model schema equivalency with optional dtypes Signed-off-by: Deepyaman Datta <[email protected]> * Support specifying and validating optional columns Signed-off-by: Deepyaman Datta <[email protected]> * Add DuckDB for testing and regenerate requirements Signed-off-by: Deepyaman Datta <[email protected]> --------- Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Register string datatype and update existing tests (#1766) Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Add missing `type` field for the `Int32` data type (#1771) Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Implement minimal `coerce` and corresponding tests (#1772) * Implement minimal `coerce` and corresponding tests Signed-off-by: Deepyaman Datta <[email protected]> * Do not exclude 3.9 in Ibis CI, and regenerate reqs Signed-off-by: Deepyaman Datta <[email protected]> --------- Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Ibis check backend (#1831) * [wip] add minimal ibis check backend implementation Signed-off-by: cosmicBboy <[email protected]> * support scalar, column, and table check output types Signed-off-by: cosmicBboy <[email protected]> * support scalar, column, and table check output types Signed-off-by: cosmicBboy <[email protected]> * Ibis check backend suggestions (#1855) * Apply suggestions from code review Signed-off-by: Deepyaman Datta <[email protected]> * Fix row-order-dependent order by adding table cols Signed-off-by: Deepyaman Datta <[email protected]> --------- Signed-off-by: Deepyaman Datta <[email protected]> * fix lint Signed-off-by: cosmicBboy <[email protected]> * fix unit tests Signed-off-by: cosmicBboy <[email protected]> --------- Signed-off-by: cosmicBboy <[email protected]> Signed-off-by: Deepyaman Datta <[email protected]> Co-authored-by: Deepyaman Datta <[email protected]> Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Remove Ibis backend's use of `scalar_failure_case` Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Update signature of `run_schema_components_checks` Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Fix violations in `pandera/backends//container.py` Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Remove use of multimethod library and align Polars Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * fix backend registration Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Implement minimal built-in checks for Ibis backend (#1885) * Implement minimal built-in checks for Ibis backend Signed-off-by: Deepyaman Datta <[email protected]> * Implement `Column` validation for the Ibis backend Signed-off-by: Deepyaman Datta <[email protected]> * Promote check object to table during preprocessing Signed-off-by: Deepyaman Datta <[email protected]> * Remove extraneous fixture for backend registration Signed-off-by: Deepyaman Datta <[email protected]> * Resolve lint (unused imports, undefined variables) Signed-off-by: Deepyaman Datta <[email protected]> * Partially standardize docstrings of builtin checks Signed-off-by: Deepyaman Datta <[email protected]> * Fix the `preprocess` docstrings copied from pandas Signed-off-by: Deepyaman Datta <[email protected]> * Format pandera/backends/ibis/checks.py using Black Signed-off-by: Deepyaman Datta <[email protected]> --------- Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Update test to reflect scalar output since Ibis 10 (#1907) Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Use `Union` type for `PysparkObject` to fix typing Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Add Ibis with the DuckDB extra as a dev dependency Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Update/align noxfile.py and generated requirements Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Supress unused argument check for `data_container` Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Include `polars` extra dep for testing Ibis dtypes Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Update check example to correctly process IbisData Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Execute validated schema to get the desired result Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Implement "ne" built-in check for the Ibis backend Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Implement int/uint/float types, except for float16 Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Implement timestamp type, and test built-in checks Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Support built-in checks for interval-typed columns Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Blacken pandera/engines/ibis_engine.py module code Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Implement `dt.Date` type, and test built-in checks Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Implement `dt.Time` type, and test built-in checks Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Implement `gt` and `ge` check for the Ibis backend Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Standardize docstrings, don't say "data container" Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Implement `lt` and `le` check for the Ibis backend Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Fix "form" to "from", and align docstring for test Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Apply suggestion from Copilot to fix error message Co-authored-by: Copilot <[email protected]> Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Support table-level checks, including for built-in Signed-off-by: Deepyaman Datta <[email protected]> Co-authored-by: cosmicBboy <[email protected]> Signed-off-by: cosmicBboy <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Implement `is_in_range` check for the Ibis backend Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Implement `isin()` built-in check for Ibis backend Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Refactor duplicated code into `_across()` function Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Implement `notin()` builtin check for Ibis backend Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Implement the `str_matches` built-in check on Ibis Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Implement `str_contains` check on the Ibis backend Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Implement `str_startswith()` check on Ibis backend Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Implement `str_endswith()` built-in check for Ibis Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Implement the `str_length` built-in check for Ibis Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Implement the unique_values_eq built-in Ibis check Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Add entry for `ibis` to the backend support matrix Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Write detailed documentation on using Ibis backend Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Replace `ibis.expr.types` imports (where possible) Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Implement `typing` module and decorators for Ibis Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Duplicate tests for Ibis decorators implementation Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Use `code-cell` over `testcode` to build Ibis docs Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Add Ibis docs to the supported libraries `toctree` Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Create element-wise checks automatically for users (#2043) * Create element-wise checks automatically for users Signed-off-by: Deepyaman Datta <[email protected]> * Mark unsupported scenario with element-wise checks Signed-off-by: Deepyaman Datta <[email protected]> --------- Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Remove `to_format` methods that don't work on Ibis (#2044) Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Execute dataframe-level checks on the Ibis backend (#2041) * Execute dataframe-level checks on the Ibis backend Signed-off-by: Deepyaman Datta <[email protected]> * Remove unnecessary, un-Pythonic array length check Signed-off-by: Deepyaman Datta <[email protected]> * Remove bit on separately running data-level checks Signed-off-by: Deepyaman Datta <[email protected]> --------- Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * Mark unsupported scenario with element-wise checks (#2046) Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * handle dataframe-level failure cases: convert row to dict Signed-off-by: cosmicBboy <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * bugfix/1927 (#2019) * fix generating pydantic core schema for generic types Signed-off-by: Jarek-Rolski <[email protected]> * test pytest for typed generic DataFrame Signed-off-by: Jarek-Rolski <[email protected]> * fix issue with docs gitactions test Signed-off-by: Jarek-Rolski <[email protected]> * fix pytest test_typed_generic_dataframe Signed-off-by: Jarek-Rolski <[email protected]> * fix pytest test_typed_generic_dataframe Signed-off-by: Jarek-Rolski <[email protected]> * improve changes Signed-off-by: Jarek-Rolski <[email protected]> * bug fix Signed-off-by: Jarek-Rolski <[email protected]> * fix pydantic tests for pydantic<2 Signed-off-by: Jarek-Rolski <[email protected]> * add pytest test_typed_dataframe_model_json_schema Signed-off-by: Jarek-Rolski <[email protected]> --------- Signed-off-by: Jarek-Rolski <[email protected]> Signed-off-by: Andrej Iring <[email protected]> * resolve failing pylint tests Signed-off-by: Andrej Iring <[email protected]> --------- Signed-off-by: Andrej Iring <[email protected]> Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: cosmicBboy <[email protected]> Signed-off-by: Jarek-Rolski <[email protected]> Co-authored-by: Niels Bantilan <[email protected]> Co-authored-by: Deepyaman Datta <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Jarek-Rolski <[email protected]>

deepyaman changed the base branch from main to ibis-dev December 25, 2024 18:40

deepyaman changed the base branch from ibis-dev to main December 25, 2024 18:40

deepyaman force-pushed the feat/ibis/builtin-checks branch 2 times, most recently from 3d0efc6 to da8490f Compare December 26, 2024 12:38

deepyaman changed the base branch from main to ibis-dev December 31, 2024 12:50

deepyaman added 2 commits December 31, 2024 07:51

Implement minimal built-in checks for Ibis backend

88ac39e

Signed-off-by: Deepyaman Datta <[email protected]>

Implement Column validation for the Ibis backend

5744a4c

Signed-off-by: Deepyaman Datta <[email protected]>

deepyaman force-pushed the feat/ibis/builtin-checks branch from da8490f to 5744a4c Compare December 31, 2024 12:51

deepyaman added 3 commits January 4, 2025 11:21

Promote check object to table during preprocessing

73ae8a6

Signed-off-by: Deepyaman Datta <[email protected]>

Remove extraneous fixture for backend registration

b175056

Signed-off-by: Deepyaman Datta <[email protected]>

Resolve lint (unused imports, undefined variables)

b97d493

Signed-off-by: Deepyaman Datta <[email protected]>

deepyaman force-pushed the feat/ibis/builtin-checks branch from 3f3f649 to b97d493 Compare January 6, 2025 14:14

deepyaman added 2 commits January 6, 2025 08:42

Partially standardize docstrings of builtin checks

c1f5eb6

Signed-off-by: Deepyaman Datta <[email protected]>

Fix the preprocess docstrings copied from pandas

e0f868d

Signed-off-by: Deepyaman Datta <[email protected]>

deepyaman marked this pull request as ready for review January 6, 2025 15:48

deepyaman requested a review from cosmicBboy January 6, 2025 15:49

Format pandera/backends/ibis/checks.py using Black

34daca5

Signed-off-by: Deepyaman Datta <[email protected]>

deepyaman force-pushed the feat/ibis/builtin-checks branch from 4698e40 to 34daca5 Compare January 9, 2025 07:23

cosmicBboy reviewed Jan 9, 2025

View reviewed changes

deepyaman merged commit d68e842 into unionai-oss:ibis-dev Jan 13, 2025
155 checks passed

deepyaman deleted the feat/ibis/builtin-checks branch January 15, 2025 07:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Implement minimal built-in checks for Ibis backend #1885

Implement minimal built-in checks for Ibis backend #1885

Uh oh!

deepyaman commented Dec 25, 2024 •

edited

Loading

Uh oh!

codecov bot commented Dec 25, 2024 •

edited

Loading

Uh oh!

cosmicBboy left a comment •

edited

Loading

Uh oh!

deepyaman commented Jan 13, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Implement minimal built-in checks for Ibis backend #1885

Implement minimal built-in checks for Ibis backend #1885

Uh oh!

Conversation

deepyaman commented Dec 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Dec 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

cosmicBboy left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deepyaman commented Jan 13, 2025

Uh oh!

Uh oh!

Uh oh!

deepyaman commented Dec 25, 2024 •

edited

Loading

codecov bot commented Dec 25, 2024 •

edited

Loading

cosmicBboy left a comment •

edited

Loading